Video interpolation is used to generate one or more additional frames between two consecutive frames. Increasing the number of frames enables video playback at a higher frame rate and slow motion playback. Video interpolation may also be used to generate smooth view transitions. The intermediate frames that are generated should be coherent, both spatially and temporally so that artifacts caused by occlusion are minimized. Conventional techniques generate a single frame between two consecutive frames and, when large motions occur between the two consecutive frames, artifacts are present in the single frame.
The conventional techniques cannot be directly used to generate arbitrary higher frame-rate videos. An approach of simply applying a single frame video interpolation method recursively to generate multiple intermediate frames has at least two limitations. First, recursive single-frame interpolation cannot be fully parallelized, and is therefore slow, since some frames cannot be computed until other frames are finished (e.g., in seven-frame interpolation, frame 2 depends on 0 and 4, while frame 4 depends on 0 and 8). Errors also accumulate during recursive interpolation. Second, single-frame interpolation can only generate 2n−1 intermediate frames (e.g., 3, 7). As a result, recursively applying single-frame video interpolation to generate 1008-fps (frame-per-second) video from 24-fps video requires generating 41 intermediate frames which is not efficient.
More recently, optical flow techniques have been used to perform frame interpolation. However, motion boundaries and severe occlusions are still challenging, and thus the interpolated frames tend to have artifacts around boundaries of moving objects. There is a need for addressing these issues and/or other issues associated with the prior art.